{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Tce3stUlHN0L"
      },
      "source": [
        "##### Copyright 2018 The TensorFlow Authors.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "cellView": "form",
        "colab": {},
        "colab_type": "code",
        "id": "tuOe1ymfHZPu"
      },
      "outputs": [],
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "MfBg1C5NB3X0"
      },
      "source": [
        "# Use TPUs\n",
        "\n",
        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://www.tensorflow.org/guide/tpu\"><img src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" />View on TensorFlow.org</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/guide/tpu.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://github.com/tensorflow/docs/blob/master/site/en/guide/tpu.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a href=\"https://storage.googleapis.com/tensorflow_docs/docs/site/en/guide/tpu.ipynb\"><img src=\"https://www.tensorflow.org/images/download_logo_32px.png\" />Download notebook</a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Ys81cOhXOWUP"
      },
      "source": [
        "Experimental support for Cloud TPUs is currently available for Keras and Google Colab. Before you run this Colab notebooks, ensure that your hardware accelerator is a TPU by checking your notebook settings: Runtime > Change runtime type > Hardware accelerator > TPU."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "Cw0WRaChRxTL"
      },
      "outputs": [],
      "source": [
        "import tensorflow as tf\n",
        "\n",
        "import os\n",
        "import tensorflow_datasets as tfds"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "yDWaRxSpwBN1"
      },
      "source": [
        "## TPU Initialization\n",
        "TPUs are usually on Cloud TPU workers which are different from the local process running the user python program. Thus some initialization work needs to be done to connect to the remote cluster and initialize TPUs. Note that the `tpu` argument to `TPUClusterResolver` is a special address just for Colab. In the case that you are running on Google Compute Engine (GCE), you should instead pass in the name of your CloudTPU."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "dCqWMqvtwOLs"
      },
      "source": [
        "Note: The TPU initialization code has to be at the beginning of your program."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "dKPqF8d1wJCV"
      },
      "outputs": [],
      "source": [
        "resolver = tf.distribute.cluster_resolver.TPUClusterResolver(tpu='grpc://' + os.environ['COLAB_TPU_ADDR'])\n",
        "tf.config.experimental_connect_to_cluster(resolver)\n",
        "# This is the TPU initialization code that has to be at the beginning.\n",
        "tf.tpu.experimental.initialize_tpu_system(resolver)\n",
        "print(\"All devices: \", tf.config.list_logical_devices('TPU'))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Mv7kehTZ1Lq_"
      },
      "source": [
        "## Manual device placement\n",
        "After the TPU is initialized, you can use manual device placement to place the computation on a single TPU device.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "XRZ4kMoxBNND"
      },
      "outputs": [],
      "source": [
        "a = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]])\n",
        "b = tf.constant([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]])\n",
        "with tf.device('/TPU:0'):\n",
        "  c = tf.matmul(a, b)\n",
        "print(\"c device: \", c.device)\n",
        "print(c)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "_NJm-kgFO0cC"
      },
      "source": [
        "## Distribution strategies\n",
        "Most times users want to run the model on multiple TPUs in a data parallel way. A distribution strategy is an abstraction that can be used to drive models on CPU, GPUs or TPUs. Simply swap out the distribution strategy and the model will run on the given device. See the [distribution strategy guide](./distributed_training.ipynb) for more information."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "DcDPMZs-9uLJ"
      },
      "source": [
        "First, creates the `TPUStrategy` object."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "7SO23K8oRpjI"
      },
      "outputs": [],
      "source": [
        "strategy = tf.distribute.experimental.TPUStrategy(resolver)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "JlaAmswWPsU6"
      },
      "source": [
        "To replicate a computation so it can run in all TPU cores, you can simply pass it to `strategy.run` API. Below is an example that all the cores will obtain the same inputs `(a, b)`, and do the matmul on each core independently. The outputs will be the values from all the replicas."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "-90CL5uFPTOa"
      },
      "outputs": [],
      "source": [
        "@tf.function\n",
        "def matmul_fn(x, y):\n",
        "  z = tf.matmul(x, y)\n",
        "  return z\n",
        "\n",
        "z = strategy.run(matmul_fn, args=(a, b))\n",
        "print(z)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "uxgYl6kGHJLc"
      },
      "source": [
        "## Classification on TPUs\n",
        "As we have learned the basic concepts, it is time to look at a more concrete example. This guide demonstrates how to use the distribution strategy `tf.distribute.experimental.TPUStrategy` to drive a Cloud TPU and train a Keras model.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "gKRALGgt_kCo"
      },
      "source": [
        "### Define a Keras model\n",
        "Below is the definition of MNIST model using Keras, unchanged from what you would use on CPU or GPU. Note that Keras model creation needs to be inside `strategy.scope`, so the variables can be created on each TPU device. Other parts of the code is not necessary to be inside the strategy scope."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "DiBiN-Z_R7P7"
      },
      "outputs": [],
      "source": [
        "def create_model():\n",
        "  return tf.keras.Sequential(\n",
        "      [tf.keras.layers.Conv2D(256, 3, activation='relu', input_shape=(28, 28, 1)),\n",
        "       tf.keras.layers.Conv2D(256, 3, activation='relu'),\n",
        "       tf.keras.layers.Flatten(),\n",
        "       tf.keras.layers.Dense(256, activation='relu'),\n",
        "       tf.keras.layers.Dense(128, activation='relu'),\n",
        "       tf.keras.layers.Dense(10)])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "qYOYjYTg_31l"
      },
      "source": [
        "### Input datasets\n",
        "Efficient use of the `tf.data.Dataset` API is critical when using a Cloud TPU, as it is impossible to use the Cloud TPUs unless you can feed them data quickly enough. See [Input Pipeline Performance Guide](./data_performance.ipynb) for details on dataset performance.\n",
        "\n",
        "For all but the simplest experimentation (using `tf.data.Dataset.from_tensor_slices` or other in-graph data) you will need to store all data files read by the Dataset in Google Cloud Storage (GCS) buckets.\n",
        "\n",
        "For most use-cases, it is recommended to convert your data into `TFRecord` format and use a `tf.data.TFRecordDataset` to read it. See [TFRecord and tf.Example tutorial](../tutorials/load_data/tfrecord.ipynb) for details on how to do this. This, however, is not a hard requirement and you can use other dataset readers (`FixedLengthRecordDataset` or `TextLineDataset`) if you prefer.\n",
        "\n",
        "Small datasets can be loaded entirely into memory using `tf.data.Dataset.cache`.\n",
        "\n",
        "Regardless of the data format used, it is strongly recommended that you use large files, on the order of 100MB. This is especially important in this networked setting as the overhead of opening a file is significantly higher.\n",
        "\n",
        "Here you should use the `tensorflow_datasets` module to get a copy of the MNIST training data. Note that `try_gcs` is specified to use a copy that is available in a public GCS bucket. If you don't specify this, the TPU will not be able to access the data that is downloaded. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "noAd416KSCo7"
      },
      "outputs": [],
      "source": [
        "def get_dataset(batch_size, is_training=True):\n",
        "  split = 'train' if is_training else 'test'\n",
        "  dataset, info = tfds.load(name='mnist', split=split, with_info=True,\n",
        "                            as_supervised=True, try_gcs=True)\n",
        "\n",
        "  def scale(image, label):\n",
        "    image = tf.cast(image, tf.float32)\n",
        "    image /= 255.0\n",
        "\n",
        "    return image, label\n",
        "\n",
        "  dataset = dataset.map(scale)\n",
        "\n",
        "  # Only shuffle and repeat the dataset in training. The advantage to have a\n",
        "  # infinite dataset for training is to avoid the potential last partial batch\n",
        "  # in each epoch, so users don't need to think about scaling the gradients\n",
        "  # based on the actual batch size.\n",
        "  if is_training:\n",
        "    dataset = dataset.shuffle(10000)\n",
        "    dataset = dataset.repeat()\n",
        "\n",
        "  dataset = dataset.batch(batch_size)\n",
        "\n",
        "  return dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "mgUC6A-zCMEr"
      },
      "source": [
        "### Train a model using Keras high level APIs\n",
        "\n",
        "You can train a model simply with Keras fit/compile APIs. Nothing here is TPU specific, you would write the same code below if you had mutliple GPUs and where using a `MirroredStrategy` rather than a `TPUStrategy`. To learn more, check out the [Distributed training with Keras](https://www.tensorflow.org/tutorials/distribute/keras) tutorial."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "ubmDchPqSIx0"
      },
      "outputs": [],
      "source": [
        "with strategy.scope():\n",
        "  model = create_model()\n",
        "  model.compile(optimizer='adam',\n",
        "                loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n",
        "                metrics=['sparse_categorical_accuracy'])\n",
        "\n",
        "batch_size = 200\n",
        "steps_per_epoch = 60000 // batch_size\n",
        "\n",
        "train_dataset = get_dataset(batch_size, is_training=True)\n",
        "test_dataset = get_dataset(batch_size, is_training=False)\n",
        "\n",
        "model.fit(train_dataset,\n",
        "          epochs=5,\n",
        "          steps_per_epoch=steps_per_epoch,\n",
        "          validation_data=test_dataset)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "0rRALBZNCO4A"
      },
      "source": [
        "### Train a model using custom training loop.\n",
        "You can also create and train your models using `tf.function` and `tf.distribute` APIs directly. `strategy.experimental_distribute_datasets_from_function` API is used to distribute the dataset given a dataset function. Note that the batch size passed into the dataset will be per replica batch size instead of global batch size in this case. To learn more, check out the [Custom training with tf.distribute.Strategy](https://www.tensorflow.org/tutorials/distribute/custom_training) tutorial.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "DxdgXPAL6iFE"
      },
      "source": [
        "First, create the model, datasets and tf.functions."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "9aHhqwao2Fxi"
      },
      "outputs": [],
      "source": [
        "# Create the model, optimizer and metrics inside strategy scope, so that the\n",
        "# variables can be mirrored on each device.\n",
        "with strategy.scope():\n",
        "  model = create_model()\n",
        "  optimizer = tf.keras.optimizers.Adam()\n",
        "  training_loss = tf.keras.metrics.Mean('training_loss', dtype=tf.float32)\n",
        "  training_accuracy = tf.keras.metrics.SparseCategoricalAccuracy(\n",
        "      'training_accuracy', dtype=tf.float32)\n",
        "\n",
        "# Calculate per replica batch size, and distribute the datasets on each TPU\n",
        "# worker.\n",
        "per_replica_batch_size = batch_size // strategy.num_replicas_in_sync\n",
        "\n",
        "train_dataset = strategy.experimental_distribute_datasets_from_function(\n",
        "    lambda _: get_dataset(per_replica_batch_size, is_training=True))\n",
        "\n",
        "@tf.function\n",
        "def train_step(iterator):\n",
        "  \"\"\"The step function for one training step\"\"\"\n",
        "\n",
        "  def step_fn(inputs):\n",
        "    \"\"\"The computation to run on each TPU device.\"\"\"\n",
        "    images, labels = inputs\n",
        "    with tf.GradientTape() as tape:\n",
        "      logits = model(images, training=True)\n",
        "      loss = tf.keras.losses.sparse_categorical_crossentropy(\n",
        "          labels, logits, from_logits=True)\n",
        "      loss = tf.nn.compute_average_loss(loss, global_batch_size=batch_size)\n",
        "    grads = tape.gradient(loss, model.trainable_variables)\n",
        "    optimizer.apply_gradients(list(zip(grads, model.trainable_variables)))\n",
        "    training_loss.update_state(loss * strategy.num_replicas_in_sync)\n",
        "    training_accuracy.update_state(labels, logits)\n",
        "\n",
        "  strategy.run(step_fn, args=(next(iterator),))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Ibi7Z97V6xsQ"
      },
      "source": [
        "Then run the training loop."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "1du5cXWt6Vtw"
      },
      "outputs": [],
      "source": [
        "steps_per_eval = 10000 // batch_size\n",
        "\n",
        "train_iterator = iter(train_dataset)\n",
        "for epoch in range(5):\n",
        "  print('Epoch: {}/5'.format(epoch))\n",
        "\n",
        "  for step in range(steps_per_epoch):\n",
        "    train_step(train_iterator)\n",
        "  print('Current step: {}, training loss: {}, accuracy: {}%'.format(\n",
        "      optimizer.iterations.numpy(),\n",
        "      round(float(training_loss.result()), 4),\n",
        "      round(float(training_accuracy.result()) * 100, 2)))\n",
        "  training_loss.reset_states()\n",
        "  training_accuracy.reset_states()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "TnZJUM3qIjKu"
      },
      "source": [
        "### Improving performance by multiple steps within `tf.function`\n",
        "The performance can be improved by running multiple steps within a `tf.function`. This is achieved by wrapping the `strategy.run` call with a `tf.range` inside `tf.function`, AutoGraph will convert it to a `tf.while_loop` on the TPU worker.\n",
        "\n",
        "Although with better performance, there are tradeoffs comparing with a single step inside `tf.function`. Running multiple steps in a `tf.function` is less flexible, you cannot run things eagerly or arbitrary python code within the steps.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "2grYvXLzJYkP"
      },
      "outputs": [],
      "source": [
        "@tf.function\n",
        "def train_multiple_steps(iterator, steps):\n",
        "  \"\"\"The step function for one training step\"\"\"\n",
        "\n",
        "  def step_fn(inputs):\n",
        "    \"\"\"The computation to run on each TPU device.\"\"\"\n",
        "    images, labels = inputs\n",
        "    with tf.GradientTape() as tape:\n",
        "      logits = model(images, training=True)\n",
        "      loss = tf.keras.losses.sparse_categorical_crossentropy(\n",
        "          labels, logits, from_logits=True)\n",
        "      loss = tf.nn.compute_average_loss(loss, global_batch_size=batch_size)\n",
        "    grads = tape.gradient(loss, model.trainable_variables)\n",
        "    optimizer.apply_gradients(list(zip(grads, model.trainable_variables)))\n",
        "    training_loss.update_state(loss * strategy.num_replicas_in_sync)\n",
        "    training_accuracy.update_state(labels, logits)\n",
        "\n",
        "  for _ in tf.range(steps):\n",
        "    strategy.run(step_fn, args=(next(iterator),))\n",
        "\n",
        "# Convert `steps_per_epoch` to `tf.Tensor` so the `tf.function` won't get \n",
        "# retraced if the value changes.\n",
        "train_multiple_steps(train_iterator, tf.convert_to_tensor(steps_per_epoch))\n",
        "\n",
        "print('Current step: {}, training loss: {}, accuracy: {}%'.format(\n",
        "      optimizer.iterations.numpy(),\n",
        "      round(float(training_loss.result()), 4),\n",
        "      round(float(training_accuracy.result()) * 100, 2)))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "WBKVhMvWjibf"
      },
      "source": [
        "## Next steps\n",
        "\n",
        "*   [Google Cloud TPU Documentation](https://cloud.google.com/tpu/docs/) - Set up and run a Google Cloud TPU.\n",
        "*   [Distributed training with TensorFlow](./distributed_training.ipynb) - How to use distribution strategy and links to many example showing best practices.\n",
        "*   [TensorFlow Official Models](https://github.com/tensorflow/models/tree/master/official) - Examples of state of the art TensorFlow 2.x models that are Cloud TPU compatible.\n",
        "*   [The Google Cloud TPU Performance Guide](https://cloud.google.com/tpu/docs/performance-guide) - Enhance Cloud TPU performance further by adjusting Cloud TPU configuration parameters for your application."
      ]
    }
  ],
  "metadata": {
    "accelerator": "TPU",
    "colab": {
      "collapsed_sections": [],
      "name": "tpu.ipynb",
      "private_outputs": true,
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
